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Elastic image registration 



5 The present invention relates to the field of digital imaging, for example, 

in the field of medical imaging. In particular, the present invention relates to a method 
of registering a first image (e.g. a floating image) and a second image (e.g. a reference 
image), an image processing device and to a computer program for registering a first 
image and a second image. 

10 When two images of the same object have been taken from different 

projections or at different points of time, or during different moving stages of the object 
of interest, or when two images do not come from the same object, but from objects of 
the same class (for example, two faces or two brains), it may be highly desirable to 
establish correspondences. Two classes of deformations have to be compensated in 

15 order to establish these correspondences: rigid deformations resulting from translational 
and rotational movements of the object imaged and non-rigid deformations resulting 
from elastic deformations of the object imaged. An example of such an application is to 
match a brain with an anatomical atlas in order to automatically find abnormalities or to 
label the brain with anatomical names. Another example is to observe the deformation 

20 over the time of, for example, a beating heart or to register images of a deformable 

region of a patient (for example, abdomen) taken at different times. Another example is 
the combination of images acquired with different modalities such as Computer 
Tomography (CT), Magnetic Resonance Imaging (MRI), Ultra Sound (US), Positron 
Emission Tomography (PET), Single Photon Emission Computed Tomography 

25 (SPECT) to improve the diagnostic process through the combined analysis of 
anatomical and physiological information. 

It is an object of the present invention to provide for an improved image 

30 registration. 
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According to an exemplary embodiment of the present invention as set 
forth in claim 1, the above object may be solved by a method of registering a first 
image and a second image, wherein at least one first landmark is selected in the first 
image and at least one second landmark is selected in the second image. Then, these 
5 first and second images are registered by using a similarity value, which relates to a 
similarity of a first region in the first image determined by at least one first landmark 
and a second region in the second image determined by at least second landmark. The 
first and second landmarks correspond to each other. 

In other words, according to this exemplary embodiment of the present 
10 invention, landmarks, for example in medical applications anatomical landmarks, are 
automatically or manually selected in the images. These landmarks are used to define 
corresponding regions in the images. For example, in angiopraphic images relating to 
different heart phases, cross points of the LAD (left anterior descending) and the LCX 
(left circumflex) may be used as landmarks in the different phase images. Then, a 
15 4 similarity value is determined to express a similarity between the region indicated by 
the first landmark(s) in the first image and the region indicated by the second 
■ landmark(s) in the second image. This similarity value is used to register the images. 

Corresponding landmarks in the first and second images are landmarks 
which, for example, for anatomical applications, mark the same or corresponding 
20 anatomical structures in the first or second images. I.e. corresponding landmarks in the 
first or second images are points or regions ideally to be registered on each other. 

According to another exemplary embodiment of the present invention as 
set forth in claim 2, the first and second landmarks are selected in accordance with a 
qualifying function. This means that, for example, only the most significant landmarks, 
25 i.e. only the most significant corresponding features are selected as landmarks. In other 
words, this exemplary embodiment of the present invention focuses the computational 
work on few but significant landmarks, for example, in areas where needed due to large 
and highly non-linear deformations and/or well delineated anatomical structures 
allowing for unambiguous and precise registration. It should be noted that the 
30 landmarks may be selected interactively or automatically. 

Advantageously, according to this exemplary embodiment of the present 
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invention, the speed and robustness of the image registration may be improved by 
concentrating only on significant landmarks and image areas showing highly non-linear 
deformations. 

According to another exemplary embodiment of the present invention as 
5 set forth in claim 3, a local deformation is determined for each landmark in the first and 
second images. These local deformations are expanded into a global deformation for the 
images. The global deformation may be used to perform the registration. 

According to another exemplary embodiment of the present invention as 
set forth in claim 4, a similarity value is determined to describe a similarity between 
10 respective regions in the first and second images defined by the landmarks in the first 
and second images. This is done by using the global deformation fields. 
Advantageously, by this, a similarity of defined regions in the first and second regions 
may be determined and regions may be identified where the similarity is not yet 
sufficient. 

1 5 According to another exemplary embodiment of the present invention as 

set forth in claim 5, for regions where the similarity is not yet sufficient, a new 
landmark is defined in the first and second image and the above described method may 
be iteratively repeated, i.e. for the new sub-regions defined by the new landmark, new 
sub-similarities with sub-similarity values are determined by using a local deformation 

20 field of this new landmark, which is expanded to a global deformation field for the 
image. Then, all the newly defined sub-regions are tested as to whether a respective 
similarity value meets with a pre-set threshold and if not, according to this exemplary 
embodiment of the present invention, new landmarks or new regions may be iteratively 
defined. 

25 Due to the fact that the refinement process is always limited to the region 

with insufficient similarity and is not applied to the whole images, the computational 
efforts may be reduced. Furthermore, by either automatically or interactively 
performing this iterative refinement, it allows for a very accurate registration. 

According to another exemplary embodiment of the present invention as 

30 set forth in claim 6, the method is iteratively repeated until all similarities of all regions 
and all sub-regions exceed a pre-set threshold value. 
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According to another exemplary embodiment of the present invention as 
set forth in claim 7, the method is applied in medical imaging on one of CT data sets, 
MRI data sets, PET data sets, SPECT data sets, and US data sets. 

According to another exemplary embodiment of the present invention as 
5 set forth in claim 8, an image processing device is provided, allowing to perform a xion- 
rigid image registration method by an iterative refinement process based on landmarks 
and a similarity value such as a gray value based measurement of a region in the 
images. 

Advantageously, the image processing device according to this 
10 exemplary embodiment of the present invention has an improved registration speed and . 
provides for a very accurate and robust registration of the first and second images. 

According to another exemplary embodiment of the present invention as 
set forth in claim 9, a computer program is provided for registering a first image and a 
second image. The computer program may be written in any suitable programming 
15 language, such as C++ and may be stored on a computer readable device, such as a GD- 
ROM. However, the computer program according to the present invention may also be 
presented over a network such as the World Wide Web, from which it may be 
downloaded, for example, into the internal working memory of a processor. 

20 

It may be seen as the gist of an exemplary embodiment of the present 
invention that a non-rigid image registration is performed by an iterative refinement 
process based on landmarks combining local template matching. An example for a local 
template matching is described, for example, in P. Rosch et al, "3D respiratory motion 

25 compensation by template propagation" in T. Dohi and R. Kikinis, editors, 

"Proceedings of the 5 th international conference on medical image computing and 
computer-assisted intervention" - MICCAI 2002, pages 639-646, Springer, 2002, 
which is hereby incorporated by reference. 

The image similarity in the respective regions may, for example, be 

30 determined in accordance with J. V. Hajnal et al, editors, "Medical image registration", 
CRC Press, 2001, which is hereby incorporated by reference. The method focuses the 
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computational work on few but significant landmarks and automatically refines the 
deformation field only in areas where needed due to large and highly non-linear 
deformations and/or well delineated anatomical structures allowing for unambiguous 
and precise registration. When A denotes an image that has to be registered with a 
5 reference image B, starting from a few but significant landmarks, image A is tiled with 
simplices P A spanned by d+l (d = dimensionality of the data set) neighboring 
landmarks. A global deformation field D is constructed by locating the landmarks in B 
based on local (for instance affine) deformations in their neighborhoods. Every simplex 
(region defined by the landmarks) P A is transformed into a simplex P B = D(Pa) and the 

10 similarity of, for example, the gray values of image A lying in P A and the gray values 
of image B lying in P B is estimated using, for example, mutual information or cross- 
correlation. If P A and P B are not sufficiently similar, a new landmark is located in P A 
and P A is divided into a plurality of d+ 1 smaller simplices. Advantageously, due to this, 
the tiling is iteratively refined until a desired similarity is achieved, resulting in a tiling > 

15 grid that automatically adapts itself to local non-linearities and, for example, to \ . . 
anatomical structures. 

These and other aspects of the present invention will become apparent 
from and elucidated with reference to the embodiments described hereinafter. • 

Exemplary embodiments of the present invention will be described in the 

20 following, with reference to the following drawings: i 



Fig. 1 shows a schematic representation of an image processing device 
according to an exemplary embodiment of the present invention, 
adapted to execute a method according to an exemplary 
embodiment of the present invention. 

Fig. 2 shows a simplified flow-chart of an exemplary embodiment of a 
method according to the present invention, which may be 
performed in the image processing device depicted in Fig. 1 . 

Figs. 3 to 6 show simplified sketches of images and indicated operations 
performed on the images according to an exemplary embodiment 
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of the method according to the present invention. 
Fig. 7 shows two images as examples of a global elastic registration 

during a respiratory motion of the respiratory system performed 
in accordance with the exemplary embodiment of the method 
5 according to the present invention. 



Fig. 1 depicts an exemplary embodiment of an image processing device 
according to the present invention, for executing an exemplary embodiment of a 

10 method in accordance with the present invention. The image processing device depicted 
in Fig. 1 comprises a central processing unit (CPU) or image processor 1, connected to 
a memory 2 for storing images or similarity values used or generated during the 
method. The image processor 1 may be connected to a plurality of input/output, 
network or diagnosis devices, such as an MR device or a CT device. Furthermore, the 

15 image processor 1 may be connected to an ultrasonic scanner. The image processor is 
furthermore connected to a display device 4 (for example, a computer monitor) for 
displaying information or images computed or adapted in the image processor 1 . An 
operator may interact with the image processor 1 via a keyboard 5 and/or other 
input/output devices, which are not depicted in Fig. 1 . 

20 Fig. 2 shows a flow-chart of an exemplary embodiment of a method for 

registering a first image and a second image. 

In the following, the present invention will be described with reference 
to medical imaging, where the landmarks relate to anatomical features in the images 
and/or to areas having large and highly non-linear deformations and/or well-delineated 

25 anatomical structures allowing for an unambiguous and precise registration. However, 
it should be noted that the present invention may also be applied to non-medical 
applications, for example, in material testing or quality control, where images of actual 
products are compared to reference images. Also, this technique may be used for image 
recognition systems, for example, where faces filmed are to be matched with pre- 

30 registered faces to identify the faces or persons. 

In short, the method according to this exemplary embodiment of the 
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present invention, described in the following with reference to Fig. 2 relates to a non- 
rigid image registration by an iterative refinement process based on anatomical 
landmarks combining local template matching and gray value based measurement of 
image similarity. Let image A denote a floating image which has to be registered with a 
5 reference image B. Starting from few but significant landmarks (selected interactively 
or automatically), image A is tiled with simplices P A spent by d+1 (d = dimensionality 
of the data set) neighboring landmarks. A global deformation field D is constructed by 
locating the landmarks in image B based on local (for instance affine) deformations in 
their neighborhoods. Every simplex P A is transformed into simplex P B =D (Pa) and the 
similarity of the gray values of image A lying in P A and the gray values of image B lying 
in P B is estimated using, for example, mutual information or cross-correlation.If P A and 
Pb are not sufficiently similar, a new landmark is located in P A% and P A is divided into 

smaller simplices. In this way, the tiling is iteratively refined until a desired 
similarity is achieved, resulting in a tiling grid that automatically adapts itself to local • 
non-linearities and anatomical structures. The iteration may be stopped when the 
similarity measure is sufficiently good in all simplices or when a maximum number of 
refinement steps are reached. The resulting global deformation field is then used to 
register the given images. The method is open to adjust the qualifying function for 
selecting the landmarks, the local similarity measure for matching the landmark- 
centered templates, the global similarity measure for estimating the similarity of the » 
simplices, and the type of transformation used for modeling the non-linear deformations 
to the modalities, anatomical domains and contrast mechanisms involved. This will be ■ 
described in further detail with reference to Fig. 2. 

In Fig. 2, after the start in step SI, where, for example, the images to be 
registered are acquired, in step S2 a set (a plurality) of significant landmarks {L. } is 
selected in both images A and B, which can be unambiguously assigned or allotted to 
each other, i.e. which unambiguously correspond to each other. A significance of the 
landmarks may be hereby determined by means of a pre-determined qualifying 
function. With these landmarks a simplex tiling {/>} of the images is determined. 
For a two-dimensional image, the simplices are triangles, and for a three-dimensional 
image, the simplices are tetrahedrons. This is shown in further detail in Fig. 3, depicting 
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image A or image B, where five landmarks L\ to L$ have been selected. As may be 

taken from Fig. 3, the landmarks are connected to each other by a line, such that 

simplices (here triangles) are formed, which tile the image region. This may also be 

referred to as simplex tiling. 
5 As indicated above, the selection of the landmarks L% may be performed 

automatically or interactively by an operator. For performing such landmark selection 

automatically, the above qualifying function may be applied, for example, detecting the 

brightest spots in the images or a certain gradient in the image. 

Then, in the subsequent step S3, local deformation fields A (L it Ui) are 
10 determined for each landmark Z,,. In other words, surrounding each landmark Z„ a local 

deformation field A (L it U$) is determined in the vicinity or region Ui surrounding the 

respective landmark Z,,. 

Form, size and/or dimensions of the region Ui is set such that there is 

sufficient structure, i.e. sufficient image energy in the region Ui allowing to 
15 unambiguously determine the local deformation. Furthermore, size, form and/or 

dimension of Ui is selected such that the local deformation may be sufficiently 

approximated by an affine transformation. 

This is shown in further detail in Fig. 4, showing regions U\ to U 5 
1 surrounding the respective landmarks L\ to L 5 . As may be taken from Fig. 4, the shape.: 
20 of the regions U\ to U 5 is circular. However, it should be noted that also other shapes, , 

such as rectangles may be applied. Within these regions U\ to C/ 5 , the respective local 

deformation fields A (L u U\) to D 5 (L 5> U $ ) are determined. 

In the subsequent step S4, the local deformation fields A (L it Uf) are 

expanded to a global deformation field on area L In other words, the local deformation 
25 fields A (L u Uf) are extended to a global deformation D onto the whole image area I in 

imaged. 

According to an exemplary embodiment, by varying the way the local 
deformation fields A (L u Uf) are expanded to a global deformation field A the method 
may be adapted to the respective application. For example, Gauss functions, polynomial 
30 functions, elastic functions or splines may be used to perform such expansion. 

In the subsequent step S5, a similarity S. = S(JB^ om9 D(A)\ D{Pt} ) is 
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determined. In other words, in each tile, region or part Pt of the tiling, a similarity S$ for 
expressing a similarity between the reference image i?| D(Pi) and the transformed floating 

image D(A) \ DiPf) or between the transformed reference image and the floating image. 

Advantageously, the similarity value is selected such that it increases with an increasing 
5 conformity between the images and is monotonic to a maximum value. Then, in the 
subsequent step S6 it is determined whether there is a simplex Pj with Sj< £ . 

A pre-selected value £ of the similarity value S determines in which 
parts of the tiling {/).} i.e. in which simplex, the conformity between the reference 
image B and the transformed floating image A is not yet sufficient. In case it is 
10 determined in step S6 that there is a simplex Pj with S(Pj)<T* , then, the method 

continues to step S7, where a refinement is performed. Otherwise, the method continues 
to step S8, where it ends. 

In step S7, in the simplex Pj with S(P J )<T t , i.e. in the simplex to be 
refined, a new landmark is selected. According to an exemplary embodiment of the 

15 present invention, shown in Fig. 5, the new landmark Le is selected in accordance with 

-, 

the qualifying function such that it is the most significant landmark U in the simplex Pj 
. This landmark is added to the set of landmarks }= {/,,. } vj L 6 . With the new 

landmark the simplex Pj is tiled. Here, Le is connected to each vertex (landmarks L2, 
Z, 4 , L s ) by a straight line. By this, for a two-dimensional image, three new triangles are 
20 generated and for a three-dimensional image, four new tetrahedrons are generated. In 
general: (d + 1) new simplices are generated for d special dimensions. In the set of 
simplices {P f }, the simplex Pj is replaced by the (d + 1) new simplices. Then, with the 

new set of landmarks {L £ } and set of simplices {P f } the iteration returns to step S3, 

wherein, for the pre-existing landmarks and the pre-existing regions Uu the local 
25 deformations D§ are maintained. The method iteratively continues until there is no 
simplex left with Sj < Z. 

Figs. 5 and 6 depict the further processing according to the method 
according to the present invention, as described by steps S3 to S7. In Fig. 5, the simplex 
Pj with Sj < £ is hatched. For this hatched simplex P jr defined by landmarks L% L4 and 
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L 5 , where the similarity value is less than a threshold value, a new landmark Le is 
selected. Preferably, as indicated above, the landmark L 6 is a significant or the most 
significant landmark in the region Pj. Then, by connecting Z, 6 to the landmarks L% 
and L 5 of Pp three new simplices are generated. Then, a local deformation field Z) 6 
5 (L 6 Us) is determined and the local deformation fields are expanded to a global 

deformation field. Then, similarity values are determined for each of these three new 
triangles. In case there is a triangle (here in Fig. 5 the triangle P F \ which similarity 
value is less than the pre-set threshold value, i.e. S F < £ , a new landmark L 7 is selected 
in the triangle P F as shown in Fig. 6. The method iteratively continues until the 

10 similarity value for each triangle exceeds or equals the threshold value. 

An application example of the above method is depicted in Fig. 7. In Fig. 
7, two MR images of the respiratory system are shown in an exhaled (A) (left image of 
Fig. 7); and in an inhaled state (B) (right image of Fig. 7). A registration by using the 
landmarks L\ 0 , L\\ and L\ 2 does not allow a sufficiently accurate registration of the two 

15 images since the movement of the diaphragm cannot be sufficiently reconstructed. Due 
to that, iteratively, the new significant landmark Ln has been added. 

Advantageously, the above method allows for a very accurate and 
reliable elastic registration of single or multi-modality image data sets. It may allow to 
improve speed and robustness of image registration by concentrating on significant 

20 landmarks and image areas showing highly non-linear deformations. Furthermore, it 
may allow the significant ranking and matching of landmarks, the local and global 
quality of registration and the type of non-linear transformations to be independently 
optimized with respect to the modalities, anatomical areas and expected deformations 
or application areas. It may allow to register very different imaging modalities by 

25 incorporating into these significance and quality measures a priori knowledge about the 
anatomy. Due to the focus on few but significant landmarks in conjunction with its 
automatic and spatial adaptivity to non-linear deformations, the above described 
invention may be very fast and well suited for real time registration, for example, 
during interventional procedures. In medical applications, global elastic registration 

30 may be required for a variety of clinical applications, where images acquired at 
different times, with different modalities or different patients have to be aligned. 
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Important examples of, for example, tumor diagnostics and surgery, where images of 
different modalities show different aspects of the tumor, a comparison of pre- and post- 
intervention images, analysis of time series of medical images, matching of individual 
images with anatomical atlases derived from cohort studies etc. However, it should be 
5 noted that the above described invention may also be applied to any two or three- 
dimensional data sets. 



